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Self-Driving Cars 


(aka driverless cars, autonomous cars, robocars) 


* Utopian view 
Save lives (1.3 million die every year in manual driving) 
* 4D's of human folly: drunk, drugged, distracted, drowsy driving 
* Eliminate car ownership 
* |ncrease mobility and access 
° Save money 
* Make transportation personalized, efficient, and reliable 


* Dystopian view 
* Eliminate jobs in the transportation sector 


* Failure (even if much rarer) may not depend on factors that are human 
interpretable or under human control 
* Artificial intelligence systems may be biased in ways that do not 
coincide with social norms or be ethically grounded 
° Security 
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Self-Driving Cars: Grain of Salt 


° Our intuition about what is hard or easy for Al is flawed 


* Carefully differentiate between: 
* Doubtful: Promises for future vehicles (in 2+ years) 
* Skeptical: Promises for future vehicles (in 1 year) 
* Possible: Actively testing vehicles on public roads at scale 
* Real: Available for consumer purchase today 


* Rodney brooks prediction in "My Dated Predictions": 


° >2032: A driverless "taxi" service in a major US city with arbitrary pick 
and drop off locations, even in a restricted geographical area. 


° >2045: The majority of US cities have the majority of their downtown 
under such rules. 
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Self-Driving Cars 





100 
I UU 2 


Telephone 





Av 


Electricity 









Cars 

Radio 

— Fridge 
— À 

— hir Travel 
Color TV 
Credit Card 


— Microwave 











~~ Video Games 
— 
— 7 Cell Phone 


— Internet 


&U 7% 


Adoption Rate 


7 7" Digital Camera 
— MP3 Player 


— Social Media 





— Smartphone 


0% Tablet 


1900 1910 1920 1930 1940 1950 1960 19/0 1980 1990 2000 2010 





5rjs.cn ПППППП 


| H [ mem Massachusetts ` For the full updated list of references visit: [163] MIT 6.5094: Deep Learning for Self-Driving Cars Lex Fridman January 


Institute of : 
11 Technology https://selfdrivingcars.mit.edu/references https://selfdrivingcars.mit.edu lex.mit.edu 2018 





Overview 


* Different approaches autonomy 
° Sensors 
* Companies doing it 


* Opportunities for Al and deep learning 
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Levels of Automation (SAE J3016) 


* Useful for initial discussion (especially for policy making), 
but not useful for design and engineering of the underlying 
intelligence and the holistic system performance: 


The 5 levels of driving automation 
For on-road vehicles | Human driver Automated system 











Steering and Monitoring Fallback when Automated 
acceleration/ of driving — automation system іѕ іп 
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Automated driving system 
monitors the road 











Beyond Traditional Levels: Two Al Systems 


* Starting point: 
* All cars are manually controlled until the Al system shows itself to be available 


and is elected to be turned on by the human. 
° A1: Human-Centered Autonomy 
° Definition: Al is not fully responsible 


° A2: Full Autonomy 
° Definition: Al is fully responsible 
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Beyond Traditional Levels: Two Al Systems 


Starting point: 
* All cars are manually controlled until the Al system shows itself to be available 
and is elected to be turned on by the human. 


° A1: Human-Centered Autonomy 
° Definition: Al is not fully responsible 
* Feature axis: 
° Where/how often is it "available"? (traffic, highway, sensor-based, etc.) 
* How many seconds for take-over? (0, 1, 10, etc) 
° Teleoperation support 


° A2: Full Autonomy 
° Definition: Al is fully responsible 


* Notes: 
* Noteleoperation 
* No 10-second rule: It's allowed to ask for human help, but not guaranteed to ever 
receive it. 
* Arrive to a safe destination or safe harbor. 


* Allow the human to take over when they choose to. 
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Beyond Traditional Levels: Two Al Systems 


LO * Starting point: 


* All cars are manually controlled until 
the Al system shows itself to be 
available and is elected to be turned 
on by the human. 


L1, L2, L3 * A1: Human-Centered Autonomy 
° Definition: Al is not fully responsible 


LA, L5 * A2: Full Autonomy 
* Definition: Al is fully responsible 
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Two Al Systems: 


A2: Full Autonomy 
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Two Al Systems: 


A1: Human-Centered Autonomy 
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Two Paths to an Autonomous Future 


A1: 


Human-Centered Autonomy 


* Localization and Mapping: 
Where am I? 


* Scene Understanding: 
Where/who/what/why of 
everyone else? 


* Movement Planning: 
How do I get from A to B? 


° Human-Robot Interaction: 
What is the physical and 
mental state of the driver? 


° Communicate: 
How to | convey intent to 
the driver and to the world? 


I BEN Massachusetts 
| | Institute of 
Technology 





A2: 


Full Autonomy 


Blue Text: Easier 
Red Text: Harder 


* Localization and Mapping: 
Where am I? 


* Scene Understanding: 
Where/who/what/why of 
everyone else? 


* Movement Planning: 
How do I get from A to B? 


e Human-Robot Interaction: 
What is the physical and 
mental state of the driver? 


° Communicate: 


How to | convey intent to the 
driver and to the world? 
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Is partially automated driving a bad idea? Observations from an on- 
road study 


Article - April 2018 with 447 Reads & Cite thi uw 
DOI: 10.1016/j.apergo.2017.11.010 mcm mo pem 


Victoria Banks 
ul 14.44 - University of Southampton 
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Public Perception of What Drivers Do 
in Semi-Autonomous Vehicles 
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MIT-AVT Naturalistic Driving Dataset 


St: | 275,000+ miles 
4.74 billion 
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Processed Trip Data 
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Human Behavior Shared Autonomy 


Understand Assist Share 
Behavior Behavior Control 








Large- Scale Naturalistic Data 
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MIT-AVT Naturalistic Driving Dataset 


MIT Autonomous Vehicle 
Technology Study 


Study months to-date: 21 
Participant days: 7,146 
Drivers: 78 

Vehicles: 25 

Miles driven: 275,589 
Video frames: 3.48 billion 


Study data collection is ongoing. 
Statistics updated on: Oct 23, 2017. 


Tesla Model S 
14,117 miles 
248 days in study 


Tesla Model X 
10,271 miles 
366 days in study 


Tesla Model S 
5,186 miles 
91 days in study 


Tesla Model X 
3,719 miles 
133 days in study 
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Tesla Model S 
24,657 miles 
588 days in study 


Tesla Model S 
18,666 miles 
353 days in study 


Tesla Model X 
15,074 miles 
276 days in study 


Volvo S90 
13,970 miles 
325 days in study 


Tesla Model S 
9,188 miles 
183 days in study 


Tesla Model X 
5,111 miles 


232 days in study 


Tesla Model S 
3,006 miles 
144 days in study 
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Tesla Model X 
22,001 miles 
421 days in study 


Range Rover 
Evoque 

18,130 miles 

483 days in study 


Range Rover 
Evoque 

14,499 miles 

440 days in study 


Tesla Model S 
12,353 miles 
321 days in study 


Tesla Model S 
8,319 miles 
374 days in study 


Tesla Model S 
4,596 miles 
132 days in study 


Tesla Model X 
1,306 miles 
69 days in study 
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Tesla Model S 
18,896 miles 
435 days in study 


Tesla Model S 
15,735 miles 
322 days in study 


Tesla Model S 
14,410 miles 
371 days in study 


Volvo S90 
11,072 miles 
412 days in study 


Tesla Model S 
6,720 miles 
194 days in study 


Tesla Model X 
4,587 miles 
233 days in study 


Tesla Model S 
(Offload pending) 
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500+ Miles / Day and Growing 


Mar 2016 Apr 2016 May 2016 Jun 2016 Jul 2016 Aug 2016 Sep 2016 Oct 2016 Nov 2016 Dec 2016 
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800-900 miles/day 


Data collection 
is on-going... 1000+ miles/day 
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MIT-AVT Naturalistic Driving Dataset 
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Tesla Autopilot: Patterns of Use 







Autopilot 


( 


Manual 


33.8% of the miles driven are with Autopilot engaged 
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Tesla Autopilot: Patterns of Use 


Autopilot 
83.8% 


Manual (4,844 hours) 





16.2% of the hours driven are with Autopilot engaged 
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Physical Engagement: 
Glance Classification 
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Tesla Autopilot: 
Observed Patterns of Behavior 


* Usage: People use autopilot a lot (96 miles, 96 hours) 
* Road Type: People use it on highway (using speed limit) 


* Mental Engagement: 8,000 transfers of control from 
machine show that they remain vigilant to cases when 
Autopilot creates risk. 


* Physical Engagement: Glance profile remains the same (96 
glance in manual vs autopilot by same road type) 


е The "how" of successful human-robot interaction: 


Use but Don t Trust. 


* The "why" of successful human-robot interaction: 


Learn Limitations by Exploring. 
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Self-Driving Cars: Personal Robotics View 


* First wide reaching and profound integration of persc 
In society. 


* Wide reaching: 1 billion cars on the road. 
° Profound: Human gives control of his/her life directly to robot. 


* Personal: One-on-one relationship of communication, collaboration, 
understanding and trust. 
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A self-driving car may be more a and less a perfect system. Why: 


* Flaws need humans: 
The scene understanding problem requires much more than pixel-level labeling 


* Exist with humans: 
Achieving both an enjoyable and safe driving experience may require "driving like a human". 
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Human-Centered Artificial Intelligence Approach 


30 % Needed 10 % 
Solve the perception-control And where 
problem where possible: involve the human 
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Overview 


* Different approaches autonomy 
° Sensors 
* Companies doing it 


* Opportunities for Al and deep learning 
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By 2030: Sensor Market Estimated at $36 Billion 


Institute of 
Technology 


Sensor modules market value for autonomous cars 
from 2015 to 2030 (in SB) 


35 





(in $B) 
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2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024 2025 2026 2027 2028 2029 2030 


TOTAL SENSOR MODULES IN $M 


E Dead reckoning sensors B Stereo cameras ELR radars 
E LIDARs E Cameras for surround B SR radars 
il Micro bolometers ™ Cameras for long distance E Ultra sound sensors 
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Automotive Al Sensors 


Fusion 
ECU 





Camera Radar 
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Radar 


Frame: 1 
Time: 00000 





Cheap 


Does well in extreme weather 


е Low resolution 


Most used automotive sensor for object detection and tracking 
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Velodyne LiDAR 


* Expensive 


* Extremely accurate depth 
information 


* Resolution much higher 
than radar 


° 360 degrees of visibility 





! 
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Resolution: LIDAR vs Radar 
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Camera 


* Cheap 
* Highest resolution 
* Huge data - deep learning 


e Human brains use similar sensor 
technology for driving 


* Bad at depth estimation 


* Not good in extreme weather 
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Lidar 
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Detection 
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Sensor Cost Range 


Resolution 
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Contrast 
Works in snow/ 
fog / rain 
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Ultrasonic 


Proximity 
Detection 






Sensor Cost Range 


Sensor size . Resolution 


Detects speed Works in dark 
Provides Colo | 
rovides Colour / Works in bright 
Contrast 
Works in snow/ 
fog / rain 
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Radar 


Proximity 
Detection 


Sensor Cost Range 





Sensor size Resolution 


Detects speed Works in dark 
Provides Colour / Works in bright 
Contrast 
Works in snow/ fog 
/ rain 
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Sensor Cost. д Range 


Sensor size Resolution 


Detects speed ` Works in dark 


Provides Colour / Works in bright 





Contrast 
Works in snow/ 
fog / rain 
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| Sensor size Resolution 
Resolution 


Sensor size Resolution 


Works in dark Detects speed Works in dark Detects speed Works in dark 


— Provides Colour — i 
Works in bright / Works in bright Provides Colour / Works in bright 


Contrast Contrast Contrast 
Works in snow/ Works in snow/ Works in snow/ fog 
fog / rain fog / rain / rain 
Proximity 
Detection 


Range 


Sensor size Resolution 


Detects speed Works in dark 


Provides Colour / 


Contrast Works in bright 
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fog / rain 
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Future of Sensor Technology: Camera vs LIDAR 


Proximity 
Detection 


e Radar and Ultrasonic: Sensor Cost Range 
Always there to help 


Sensor size Resolution 


Works in dark 


e Ca mera: Detects speed 
Annotated driving data grows Provides Colour / 


Contrast 


* Deeplearning algorithms improve Works in snow/ 
fog / rain 







Works in bright 


ө LIDAR: Proximity 


Detection 
ii Range Increases Sensor Cost 4 Range 
Cost drops (solid-state LIDAR) : 
Sensor size : Resolution 





Detects speed Works in dark 


Provides Col 
rovides Colour / Works in bright 
Contrast 
Works in snow/ 
fog / rain 
arisen. HTA 


| | lii mg нанета ы For the full updated list of references visit: MIT 6.5094: Deep Learning for Self-Driving Cars Lex Fridman January 


Technology https://selfdrivingcars.mit.edu/references https://selfdrivingcars.mit.edu lex.mit.edu 2018 





Overview 


* Different approaches autonomy 
° Sensors 
* Companies doing it 


* Opportunities for Al and deep learning 
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Notable: 
* April 2017: Exits testing: first rider in Phoenix 


* November 2017: 4 million miles driven autonomously 


* December 2017: No safety driver in Phoenix 
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Notable: 


° December 2017: 2 million miles driven autonomously 


5rjs.cn ПППППП 


ole For the full updated list of references visit: MIT 6.5094: Deep Learning for Self-Driving Cars Lex Fridman January 
| Technol n https://selfdrivingcars.mit.edu/references 








Notable: 


° Sep 2014: Released Autopilot 

* Oct 2016: Started Autopilot 2 from scratch. 

° Jan 2018: "1 billion miles driven in Autopilot 

° Jan 2018: ~300,000 Autopilot equipped vehicles 
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January 


Audi A8 


(Released end of 2018) 





* Thorsten Leonhardt, head of Automated Driving, Audio: 
"When the function is operated as intended, if the customer turns the traffic jam 
pilot on and uses it as intended, and the car was in control at the time of the 
accident, the driver goes to his insurance company and the insurance company will 
compensate the victims of the accident and in the aftermath they come to us and 


2) ` 
we have to pay them,” he said. 
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° Full autonomy (A2) 
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echnology 


Waymo 

Uber 

GM Cruise 
nulonomy 
OptimusRide 
Zenuity 
Voyage 


For the full updated list of references visit: 
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Notable Progress 


° Human-centered autonomy (A1) 


Tesla Autopilot - Model S/3/X 

Volvo PilotAssist - S90/XC90/XC60/V90 
Audi Traffic Jam Assist - A8 
Mercedes-Benz Drive Pilot Assist - E-Class 
Cadillac Super Cruise - CT6 

Comma.ai openpilot 
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Overview 


* Different approaches autonomy 
° Sensors 
* Companies doing it 


* Opportunities for Al and deep learning 
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Self-Driving Car Tasks 


* Localization and Mapping: 
Where am I? 
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Visual Odometry 


Up 


* 6-DOF: freed of movement 
* Changes in position: 
* Forward/backward: surge 






Back Left 
° Left/right: sway = 


* Up/down: heave Yaw 
* Orientation: 


° Pitch, Yaw, Roll 


| Forward 

Right 

° Source: Pitch 
° Monocular: | moved 1 unit 
° Stereo: | moved 1 meter 


° Mono = Stereo for far away objects 


° PS: For tiny robots everything is “far away" relative to inter-camera 
distance 


Down 
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SLAM: Simultaneous Localization and Mapping 
What works: SIFT and optical flow 
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Visual Odometry in Parts 





° (Stereo) Undistortion, Rectification 


° (Stereo) Disparity Map Computation 
* Feature Detection (e.g., SIFT, FAST) 
* Feature Tracking (e.g., KLT: Kanade-Lucas-Tomasi) 


° Trajectory Estimation 
* Use rigid parts of the scene (requires outlier/inlier detection) 


* For mono, need more info* like camera orientation and height of 
off the ground 


* Kitt, Bernd Manfred, et al. "Monocular visual odometry using a planar road model to solve scale ambiguity." (2011). 
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DeepVO: Deep Learning Based Visual Odometry 


End-to-End Method Video (Image Sequence) CNN RNN Pose 







/ / / 
/ / / 
< О / 
"2^ [— 
Ү o | tensor flow 
Á j 


“-у-” 
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DeepVO: Deep Learning Based Visual Odometry 





Ground-Truth e 
Deep-VO 


Train: 00 01 02 08 09 
Test: 05 (This video) 
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Self-Driving Car Tasks 


* Scene Understanding: 
Where is everyone else? 





»js.cn ПППППП 
| | | Ex т MIT 6.5094: Deep Learning for Self-Driving Cars Lex Fridman January 
| | Тесһпо!оду https://selfdrivingcars.mit.edu lex.mit.edu 2018 





Object Detection 





° Past approaches: cascades classifiers (Haar-like features) 


* Where deep learning can help: 
recognition, classification, detection 
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Driving Scene Segmentation 
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Road Texture and Condition from Audio 


(with Recurrent Neural Networks) 





Frequency (kHz) 


Frequency (kHz) 





— = - — — — — — - — —— _ a m a 


5 4 3 2 


1 o 1 | ' 
Current Time Offset (secs) Current Time Offset (secs) 
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Self-Driving Car Tasks 


hy, 


di N 


* Movement Planning: 
How do | get from A to B? 
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* Previous approaches: optimization-based control 


* Deep reinforcement learning: give the ability to deal with 
under-actuated control, uncertainty, motion blur, lack of sensor 
calibration or prior map information. 
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* Previous approaches: optimization-based control 


* Deep reinforcement learning: give the ability to 
deal with under-actuated control, uncertainty, 
P motion blur, lack of sensor calibration or prior 
map information. 


2 . Learning Episode 200 
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Self-Driving Car Tasks 





* Driver state: 
What's the driver up to? 
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Drive State Detection: 
A Multi-Resolutional View 


Increasing level of detection resolution and difficulty 

















Blink 
Dynamics 


Micro 
Saccades 


















Blink 
Duration 


Pupil 
Diameter 










Micro 
Glances 


Cognitive 
Load 


Frames: 1 Accuracy: 
Time: 0.03 secs 
Total Confident Decisions: 1 


Frames: 1 Accuracy: 
Time: 0.03 secs 
Total Confident Decisions: 1 
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Driver Glance Region Classification 






Latest oaze classification: 





Autopilot Status: 





кеў. W mi 


560 "м 
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Driver Body Pose Estimation 


Left: On - 7” 7 
V E On 
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Driver Emotion 





Class 1: 


Class 2: 


with Voice-Based Interaction 





Gender: Male 
Glasses: No 
Interocular distance:$182.1 
Mean Face luminance:$206.0 
= 
^ Roll angle:$2'8 
Yaw angle: 3.5 







Pitch angle: 


disgust: 
(fear) 

г іоу; 
sadness: 
surprise: 


xpressiveness: 


бепаег: 

Glasses: 

Interocular distance: 
Mean Face luminance: 
Pitch angle: 

Roll angle: 

Yaw angle: 

anger: 

contempt: 

disgust: 

fear: 


Joy: 
sadness: 
surprise: 


xpressiveness: 


[1 
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:brow furrow 
:chin raiser 
:disgust (image) 
‘eyes closed 
‘inner brow raise 


‘lip depressor 
‘lip press 
а! ‘lip pucker 
„Е ‘lip raiser 
en ‘lip suck 


:mouth open 
:nose wrinkle 
‘outer brow raise 
:smile 

:smirk 

:smirk (left) 
:smirk (right) 









:brow furrow 


Male 
Yes 
164.8 
140.5 
7.4 
—5.5 
—8.0 






“smirk (left) 
:smirk (right) 
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Cognitive Load 





Cognitive Load Estimation > dm Cognitive Load Estimation em 
Deep Learning ШТ , H A Lex Deep Learning ШТ 


DeepCogLoad 


Z m 


Cognitive Load Estimation 


.-- 
| Ñ Jack Deep Learning u j | 
DeepCogLoad 





DeepCogLoad 


d А Zë 
| å 
"A 
CONVERSATION MEDIUM CONVERSATION HIGH CONVERSATION HIGH 
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Two Paths to an Autonomous Future 


A1: 


Human-Centered Autonomy 


* Localization and Mapping: 
Where am I? 


* Scene Understanding: 
Where/who/what/why of 
everyone else? 


* Movement Planning: 
How do I get from A to B? 


° Human-Robot Interaction: 
What is the physical and 
mental state of the driver? 


° Communicate: 
How to | convey intent to 
the driver and to the world? 


I BEN Massachusetts 
| | Institute of 
Technology 





A2: 


Full Autonomy 


Blue Text: Easier 
Red Text: Harder 


* Localization and Mapping: 
Where am I? 


* Scene Understanding: 
Where/who/what/why of 
everyone else? 


* Movement Planning: 
How do I get from A to B? 


e Human-Robot Interaction: 
What is the physical and 
mental state of the driver? 


° Communicate: 


How to | convey intent to the 
driver and to the world? 
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Full Autonomy (A2) Requires Solving This 
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Full Autonomy (A2) Requires Solving This 








| gen eee MIT 6.S094: Deep Learning for Self-Driving Cars Lex Fridman January 
I | Institute of 


Technology : .mit. https://selfdrivingcars.mit.edu lex.mit.edu 2018 


Full Autonomy (A2) Requires Solving This 
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Full Autonomy (A2) Requires Solving This 
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Full Autonomy (A2) Requires a Good Reward Function 
(that balances driving safety and enjoyment) 






# 


> Du! 


S 
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Anvia Google 


O 


amazon alexa 


Autoliv 





CSRC 


Collaborative Safety Research Center 
TOYOTA 


auto 
Sms”) 


Next lecture: Deep Reinforcement Learning 


DeepTraffic 


Main Page eaderboard - About DeepTr affic 
Americans spend 8 billion hours stuck in traffic every year 


е Deep neural networks can help! 


TanesSice - 7 
а patchesAhead = 30; — 


patchesbehind < 10 
g trainIterations г 10000; 
the number of other autonoenu ehicles controlled by our network 
3 otherágents = 0; 
à u^ ts lanesside patchesAhead + patchesBehind 
D - y a N dein гем - 
0 mit Ме m 
° 0 
0 
al 0 $ 
| б 
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Road Overlay 


None 





Simulation Speed 


Fast , vehicle sk 
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